Root filesystem startup sequence
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This project is designed around the idea of a staged init sequence, which
involves several distinct executables (stages) exec'ing into each other in
context of the pid 1. Typical sequence looks like this:

	(initrd) -> startup -> supervisor -> shutdown

Here the focus is on the stages running from the real root filesystem, that
is, everything past initrd. We assume that the root filesystem has been
mounted, possibly along with some of the virtual filesystems, and the initrd
stage has exec'ed into the startup script, /etc/boot/startup.

The supervisor stage is strictly parallel, while startup and shutdown should
be mostly sequential. The sequential stages are supposed to be (msh) scripts.

The location of the shutdown scripts is fixed (/etc/boot) because of the way
they are invoked by svchub. Startup script can be placed anywhere, but for
consistency we put into /etc/boot as well.


Startup script
~~~~~~~~~~~~~~
The goal of the startup stage is to complete global kernel state setup started
by initrd, and get the system to the point where it can start the services.
Typical contents of /etc/boot/startup:

    #!/sbin/msh

    # Default environment
    setenv PATH /bin:/sbin:/usr/bin
    
    # Mount remaining virtual filesystems
    run /bin/kmount -vc /dev/pts
    run /bin/kmount -vc /run
    run /bin/kmount -vc /mnt
    run /bin/kmount -vc /tmp
    # Remount root read-write if necessary
    #run /bin/kmount -e /
    
    umask 0002
    
    # Socket location for the services, see control.txt
    mkdir /run/ctrl
    
    prctl no-new-privs
    
    exec /sbin/svchub

This part is very system-dependent, other than maybe the final exec.


Non-initrd boot
~~~~~~~~~~~~~~~
When booting directly into rootfs, the kernel will attempt to spawn /sbin/init
by default. This can be changed with init= parameter, or /sbin/init can be
a symlink to the startup script, or it can be the location for the startup
script.

There are essentially no other issues. The kernel should mount rootfs just like
an initrd would. It won't mount /dev and /sys, so appropriate lines should be
added to the startup script.


System services
~~~~~~~~~~~~~~~
A "service" is a process that does not normally exit.

Traditionally, these are known as "daemons". However, while in practice
the same entities become either "services" or "daemons" depending on how
their startup is set up (i.e. on the init system), the strict definition
of "daemon" implies double-forking which services are expected not to do.

A service can be thought of as a daemon that did not fork during startup.
Which means its pid remains valid, and the parent can wait for it.

Here the focus is mostly on the system services, that is, services started
during system startup and expected to stay alive for the whole time until
system shutdown. During this period, the process tree looks like this:

   1 svchub
    \
     +- service1
     +- service2
      \
       +- subprocess
       +- subprocess
       ...
     +- service3
      \
       +- subprocess
     +- service4
      \
       +- subprocess
       +- subprocess

Pretty much anything useful that happens in the system, happens either in
the services or in their descendent processes. The root of the tree (pid 1,
svchub) does very little, essentially just providing some information on
the running services.

The important point here is that the system must be designed to match the
structure above, with system services in the top layer of the process tree.
Most of the time, this is not a non-issue, but there are few exceptions
that may cause a bit of confusion. For instance, conventional getty is not
a service, and needs to be either spawned by, or replaced with, a service
per the definition above.


Service startup scripts
~~~~~~~~~~~~~~~~~~~~~~~
To define a system service, place an executable script into /etc/init.
Typical contents of the script, say, /etc/init/foo:

    #!/sbin/msh

    setenv FOO bar

    setgid 5
    setuid 11

    exec /usr/bin/foo

Upon startup, svchub picks up all files in /etc/init. If a service gets
added later, `svctl reload` should be used to force directory re-scan.

Services are spawned by exec()ing the script in a new-forked process.
Any process setup including setuid, prctl etc must be done in either
in the script, or in the binary being spawned.

It is important to exec() into the binary at the end of the script. For svchub,
the service is the process it spawned, that's the pid it will monitor and send
signals to. Keeping the interpreter alive at that pid is at best unnecessary,
and at worst will prevent the signals from reaching the actual service process.


Output capture and logging
~~~~~~~~~~~~~~~~~~~~~~~~~~
Services are not supposed to output anything to stdout or stderr during
normal operation. For abnormal cases, svchub captures their output into a
short ring buffer which the user can inspect using `svctl show`.

Services may use syslog (or some other similar service) for logging events
during normal operation. This is completely unrelated them being supervised
by svctl.

There is no explicit support for redirecting stdout/stderr into syslog in
this project, and doing so is discouraged.


Dependencies
~~~~~~~~~~~~
System services are expected to have no explicit dependencies, other than
the system being out of the sequential initialization stage (rootfs mounted,
kernel state set up and so on).

Explicit dependencies on devices or IPC points being available indicate
incorrectly written code, often a race condition or invalid assumptions.
Alternatively, the service in question is just not fit to be a system
service, and should be started on demand in response to some events.

# TODO: on-demand service supervisor, not in the project right now

There are some borderline cases where dependencies exists but can be worked
around by introducing timing constraints:

    #!/sbin/msh

    ...

    waitfor /dev/something

    exec /path/to/binary

In this example, svchub will spawn the script as usual, but waitfor will pause
the process for several seconds while it is still a script.


Shutdown sequence
~~~~~~~~~~~~~~~~~
When commanded to shut down, svchub kills all the services, waits for them
to dies, and execs into one of the shutdown scripts depending on how exactly
the system is shutting down:

    /etc/boot/reboot
    /etc/boot/shutdown
    /etc/boot/powerdown

    /etc/boot/failure

There may be system-dependent things there but eventually all these should
exec into `reboot` with appropriate flags. Example /etc/boot/shutdown:

    #!/sbin/msh

    exec /sbin/reboot -h

`reboot` will attempt to un-mount all remaining filesystems and invoke
sys_reboot with appropriate arguments.
